Welcome to Airflow Logging and Monitoring. After watching this video, you will be able to: Use logging capabilities to monitor the
status and diagnose problems with DAG runs. Explain accessing emitted metrics
such as counters, gauges, and timers. The logging capability is required for
developers to monitor the status of tasks in DAG runs, and to diagnose and debug issues. By default, Airflow logs are saved to local file
systems as log files. This makes it convenient for developers to quickly review the log files,
especially in a development environment. For Airflow production deployments, the
log files can be sent to cloud storage such as IBM cloud, AWS, or
Azure for remote accessing. The log files can also be sent to search
engines and dashboards for further retrieval and analysis. Airflow recommends using
Elasticsearch and Splunk, which are two popular document database and search engines,
to index, search, and analyze log files. By default, log files are organized by
DAG IDs and Task IDs, and you can find a specific log file for a task execution
using the following path convention: logs/dag_id/task_id/execution_date/try_number.log For example, suppose you want to find the log of
the first execution try of task 1 in a dummy DAG at a specific time. You can go to the folder
logs/dummy_dag/task1/execution_date/1.log and if you click the first log file, you can
check a lot of useful information, such as: the running command, command results, task results, and so on. You can also quickly review the task events
via UI provided by the Airflow webserver. You can search events with fields like
DAG ID, Task ID, and Execution Date, and quickly get an overview of the specific
DAGs and tasks you are looking for. Airflow produces three different types of metrics
for checking and monitoring components’ health. These are: counters, gauges, and timers. Counters are metrics that
will always be increasing, such as the total counts of
successful or failed tasks. Gauges are metrics that
may fluctuate. For example, the number of currently running
tasks, or DAG bag sizes. Timers are metrics related to
time duration. For instance, the time to finish a task, or the time for
a task to reach a success or failed state. Similar to logs, the metrics produced
in Airflow production deployments should be sent and analyzed by
dedicated repositories and tools. Airflow recommends using StatsD, which
is a network daemon that can gather metrics from Airflow and send them to a
dedicated metrics monitoring system. For metrics monitoring and analysis,
Airflow recommends using Prometheus, which is a popular metrics
monitoring and analysis system. Prometheus can also aggregate and visualize metrics in a dashboard for more
interactive visual analytics. In this video, you learned that: You can save Airflow logs into local file systems and send them to cloud storage,
search engines, and log analyzers. Airflow recommends sending production deployment
logs to be analyzed by Elasticsearch or Splunk. With Airflow’s UI, you can view DAGs and
task events easily. You also learned that The three types of Airflow metrics
are counters, gauges, and timers. Airflow recommends that you
send production deployment metrics for analysis by Prometheus via StatsD.